Perspectives on the role of industry in informatics research and authorship.

OBJECTIVES: Advances in informatics research come from academic, nonprofit, and for-profit industry organizations, and from academic-industry partnerships. While scientific studies of commercial products may offer critical lessons for the field, manuscripts authored by industry scientists are sometimes categorically rejected. We review historical context, community perceptions, and guidelines on informatics authorship. PROCESS: We convened an expert panel at the American Medical Informatics Association 2022 Annual Symposium to explore the role of industry in informatics research and authorship with community input. The panel summarized session themes and prepared recommendations. CONCLUSIONS: Authorship for informatics research, regardless of affiliation, should be determined by International Committee of Medical Journal Editors uniform requirements for authorship. All authors meeting criteria should be included, and categorical rejection based on author affiliation is unethical. Informatics research should be evaluated based on its scientific rigor; all sources of bias and conflicts of interest should be addressed through disclosure and, when possible, methodological mitigation.

Patient Safety Indicators During the Initial COVID-19 Pandemic Surge in the United States.

OBJECTIVE: The COVID-19 pandemic presented a challenge to inpatient safety. It is unknown whether there were spillover effects due to COVID-19 into non-COVID-19 care and safety. We sought to evaluate the changes in inpatient Agency for Healthcare Research and Quality patient safety indicators (PSIs) in the United States before and during the first surge of the pandemic among patients admitted without COVID-19. METHODS: We analyzed trends in PSIs from January 2019 to June 2020 in patients without COVID-19 using data from IBM MarketScan Commercial Database. We included members of employer-sponsored or Medicare supplemental health plans with inpatient, non-COVID-19 admissions. The primary outcomes were risk-adjusted composite and individual PSIs. RESULTS: We analyzed 1,869,430 patients admitted without COVID-19. Among patients without COVID-19, the composite PSI score was not significantly different when comparing the first surge (Q2 2020) to the prepandemic period (e.g., Q2 2020 score of 2.46 [95% confidence interval {CI}, 2.34-2.58] versus Q1 2020 score of 2.37 [95% CI, 2.27-2.46]; P = 0.22). Individual PSIs for these patients during Q2 2020 were also not significantly different, except in-hospital fall with hip fracture (e.g., Q2 2020 was 3.42 [95% CI, 3.34-3.49] versus Q4 2019 was 2.45 [95% CI, 2.40-2.50]; P = 0.01). CONCLUSIONS: The first surge of COVID-19 was not associated with worse inpatient safety for patients without COVID-19, highlighting the ability of the healthcare system to respond to the initial surge of the pandemic.

Correction: Performance of a Web-Based Reference Database With Natural Language Searching Capabilities: Usability Evaluation of DynaMed and Micromedex With Watson.

[This corrects the article DOI: 10.2196/43960.].

Performance of a Web-Based Reference Database With Natural Language Searching Capabilities: Usability Evaluation of DynaMed and Micromedex With Watson.

BACKGROUND: Evidence-based point-of-care information (POCI) tools can facilitate patient safety and care by helping clinicians to answer disease state and drug information questions in less time and with less effort. However, these tools may also be visually challenging to navigate or lack the comprehensiveness needed to sufficiently address a medical issue. OBJECTIVE: This study aimed to collect clinicians' feedback and directly observe their use of the combined POCI tool DynaMed and Micromedex with Watson, now known as DynaMedex. EBSCO partnered with IBM Watson Health, now known as Merative, to develop the combined tool as a resource for clinicians. We aimed to identify areas for refinement based on participant feedback and examine participant perceptions to inform further development. METHODS: Participants (N=43) within varying clinical roles and specialties were recruited from Brigham and Women's Hospital and Massachusetts General Hospital in Boston, Massachusetts, United States, between August 10, 2021, and December 16, 2021, to take part in usability sessions aimed at evaluating the efficiency and effectiveness of, as well as satisfaction with, the DynaMed and Micromedex with Watson tool. Usability testing methods, including think aloud and observations of user behavior, were used to identify challenges regarding the combined tool. Data collection included measurements of time on task; task ease; satisfaction with the answer; posttest feedback on likes, dislikes, and perceived reliability of the tool; and interest in recommending the tool to a colleague. RESULTS: On a 7-point Likert scale, pharmacists rated ease (mean 5.98, SD 1.38) and satisfaction (mean 6.31, SD 1.34) with the combined POCI tool higher than the physicians, nurse practitioner, and physician's assistants (ease: mean 5.57, SD 1.64, and satisfaction: mean 5.82, SD 1.60). Pharmacists spent longer (mean 2 minutes, 26 seconds, SD 1 minute, 41 seconds) on average finding an answer to their question than the physicians, nurse practitioner, and physician's assistants (mean 1 minute, 40 seconds, SD 1 minute, 23 seconds). CONCLUSIONS: Overall, the tool performed well, but this usability evaluation identified multiple opportunities for improvement that would help inexperienced users.

The Association of the First Surge of the COVID-19 Pandemic with the High- and Low-Value Outpatient Care Delivered to Adults in the USA.

BACKGROUND: The first surge of the COVID-19 pandemic entirely altered healthcare delivery. Whether this also altered the receipt of high- and low-value care is unknown. OBJECTIVE: To test the association between the April through June 2020 surge of COVID-19 and various high- and low-value care measures to determine how the delivery of care changed. DESIGN: Difference in differences analysis, examining the difference in quality measures between the April through June 2020 surge quarter and the January through March 2020 quarter with the same 2 quarters' difference the year prior. PARTICIPANTS: Adults in the MarketScan® Commercial Database and Medicare Supplemental Database. MAIN MEASURES: Fifteen low-value and 16 high-value quality measures aggregated into 8 clinical quality composites (4 of these low-value). KEY RESULTS: We analyzed 9,352,569 adults. Mean age was 44 years (SD, 15.03), 52% were female, and 75% were employed. Receipt of nearly every type of low-value care decreased during the surge. For example, low-value cancer screening decreased 0.86% (95% CI, -1.03 to -0.69). Use of opioid medications for back and neck pain (DiD +0.94 [95% CI, +0.82 to +1.07]) and use of opioid medications for headache (DiD +0.38 [95% CI, 0.07 to 0.69]) were the only two measures to increase. Nearly all high-value care measures also decreased. For example, high-value diabetes care decreased 9.75% (95% CI, -10.79 to -8.71). CONCLUSIONS: The first COVID-19 surge was associated with receipt of less low-value care and substantially less high-value care for most measures, with the notable exception of increases in low-value opioid use.

Physicians' Perceptions of and Satisfaction With Artificial Intelligence in Cancer Treatment: A Clinical Decision Support System Experience and Implications for Low-Middle-Income Countries.

As technology continues to improve, health care systems have the opportunity to use a variety of innovative tools for decision-making, including artificial intelligence (AI) applications. However, there has been little research on the feasibility and efficacy of integrating AI systems into real-world clinical practice, especially from the perspectives of clinicians who use such tools. In this paper, we review physicians' perceptions of and satisfaction with an AI tool, Watson for Oncology, which is used for the treatment of cancer. Watson for Oncology has been implemented in several different settings, including Brazil, China, India, South Korea, and Mexico. By focusing on the implementation of an AI-based clinical decision support system for oncology, we aim to demonstrate how AI can be both beneficial and challenging for cancer management globally and particularly for low-middle-income countries. By doing so, we hope to highlight the need for additional research on user experience and the unique social, cultural, and political barriers to the successful implementation of AI in low-middle-income countries for cancer care.

Effectiveness of Contact Tracing for Viral Disease Mitigation and Suppression: Evidence-Based Review.

BACKGROUND: Contact tracing in association with quarantine and isolation is an important public health tool to control outbreaks of infectious diseases. This strategy has been widely implemented during the current COVID-19 pandemic. The effectiveness of this nonpharmaceutical intervention is largely dependent on social interactions within the population and its combination with other interventions. Given the high transmissibility of SARS-CoV-2, short serial intervals, and asymptomatic transmission patterns, the effectiveness of contact tracing for this novel viral agent is largely unknown. OBJECTIVE: This study aims to identify and synthesize evidence regarding the effectiveness of contact tracing on infectious viral disease outcomes based on prior scientific literature. METHODS: An evidence-based review was conducted to identify studies from the PubMed database, including preprint medRxiv server content, related to the effectiveness of contact tracing in viral outbreaks. The search dates were from database inception to July 24, 2020. Outcomes of interest included measures of incidence, transmission, hospitalization, and mortality. RESULTS: Out of 159 unique records retrieved, 45 (28.3%) records were reviewed at the full-text level, and 24 (15.1%) records met all inclusion criteria. The studies included utilized mathematical modeling (n=14), observational (n=8), and systematic review (n=2) approaches. Only 2 studies considered digital contact tracing. Contact tracing was mostly evaluated in combination with other nonpharmaceutical interventions and/or pharmaceutical interventions. Although some degree of effectiveness in decreasing viral disease incidence, transmission, and resulting hospitalizations and mortality was observed, these results were highly dependent on epidemic severity (R0 value), number of contacts traced (including presymptomatic and asymptomatic cases), timeliness, duration, and compliance with combined interventions (eg, isolation, quarantine, and treatment). Contact tracing effectiveness was particularly limited by logistical challenges associated with increased outbreak size and speed of infection spread. CONCLUSIONS: Timely deployment of contact tracing strategically layered with other nonpharmaceutical interventions could be an effective public health tool for mitigating and suppressing infectious outbreaks by decreasing viral disease incidence, transmission, and resulting hospitalizations and mortality.

Leveraging artificial intelligence for pandemic preparedness and response: a scoping review to identify key use cases.

Artificial intelligence (AI) represents a valuable tool that could be widely used to inform clinical and public health decision-making to effectively manage the impacts of a pandemic. The objective of this scoping review was to identify the key use cases for involving AI for pandemic preparedness and response from the peer-reviewed, preprint, and grey literature. The data synthesis had two parts: an in-depth review of studies that leveraged machine learning (ML) techniques and a limited review of studies that applied traditional modeling approaches. ML applications from the in-depth review were categorized into use cases related to public health and clinical practice, and narratively synthesized. One hundred eighty-three articles met the inclusion criteria for the in-depth review. Six key use cases were identified: forecasting infectious disease dynamics and effects of interventions; surveillance and outbreak detection; real-time monitoring of adherence to public health recommendations; real-time detection of influenza-like illness; triage and timely diagnosis of infections; and prognosis of illness and response to treatment. Data sources and types of ML that were useful varied by use case. The search identified 1167 articles that reported on traditional modeling approaches, which highlighted additional areas where ML could be leveraged for improving the accuracy of estimations or projections. Important ML-based solutions have been developed in response to pandemics, and particularly for COVID-19 but few were optimized for practical application early in the pandemic. These findings can support policymakers, clinicians, and other stakeholders in prioritizing research and development to support operationalization of AI for future pandemics.

Accuracy of an Artificial Intelligence System for Cancer Clinical Trial Eligibility Screening: Retrospective Pilot Study.

BACKGROUND: Screening patients for eligibility for clinical trials is labor intensive. It requires abstraction of data elements from multiple components of the longitudinal health record and matching them to inclusion and exclusion criteria for each trial. Artificial intelligence (AI) systems have been developed to improve the efficiency and accuracy of this process. OBJECTIVE: This study aims to evaluate the ability of an AI clinical decision support system (CDSS) to identify eligible patients for a set of clinical trials. METHODS: This study included the deidentified data from a cohort of patients with breast cancer seen at the medical oncology clinic of an academic medical center between May and July 2017 and assessed patient eligibility for 4 breast cancer clinical trials. CDSS eligibility screening performance was validated against manual screening. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for eligibility determinations were calculated. Disagreements between manual screeners and the CDSS were examined to identify sources of discrepancies. Interrater reliability between manual reviewers was analyzed using Cohen (pairwise) and Fleiss (three-way) κ, and the significance of differences was determined by Wilcoxon signed-rank test. RESULTS: In total, 318 patients with breast cancer were included. Interrater reliability for manual screening ranged from 0.60-0.77, indicating substantial agreement. The overall accuracy of breast cancer trial eligibility determinations by the CDSS was 87.6%. CDSS sensitivity was 81.1% and specificity was 89%. CONCLUSIONS: The AI CDSS in this study demonstrated accuracy, sensitivity, and specificity of greater than 80% in determining the eligibility of patients for breast cancer clinical trials. CDSSs can accurately exclude ineligible patients for clinical trials and offer the potential to increase screening efficiency and accuracy. Additional research is needed to explore whether increased efficiency in screening and trial matching translates to improvements in trial enrollment, accruals, feasibility assessments, and cost.

The potential of artificial intelligence to improve patient safety: a scoping review.

Artificial intelligence (AI) represents a valuable tool that could be used to improve the safety of care. Major adverse events in healthcare include: healthcare-associated infections, adverse drug events, venous thromboembolism, surgical complications, pressure ulcers, falls, decompensation, and diagnostic errors. The objective of this scoping review was to summarize the relevant literature and evaluate the potential of AI to improve patient safety in these eight harm domains. A structured search was used to query MEDLINE for relevant articles. The scoping review identified studies that described the application of AI for prediction, prevention, or early detection of adverse events in each of the harm domains. The AI literature was narratively synthesized for each domain, and findings were considered in the context of incidence, cost, and preventability to make projections about the likelihood of AI improving safety. Three-hundred and ninety-two studies were included in the scoping review. The literature provided numerous examples of how AI has been applied within each of the eight harm domains using various techniques. The most common novel data were collected using different types of sensing technologies: vital sign monitoring, wearables, pressure sensors, and computer vision. There are significant opportunities to leverage AI and novel data sources to reduce the frequency of harm across all domains. We expect AI to have the greatest impact in areas where current strategies are not effective, and integration and complex analysis of novel, unstructured data are necessary to make accurate predictions; this applies specifically to adverse drug events, decompensation, and diagnostic errors.

Recommendations for the safe, effective use of adaptive CDS in the US healthcare system: an AMIA position paper.

The development and implementation of clinical decision support (CDS) that trains itself and adapts its algorithms based on new data-here referred to as Adaptive CDS-present unique challenges and considerations. Although Adaptive CDS represents an expected progression from earlier work, the activities needed to appropriately manage and support the establishment and evolution of Adaptive CDS require new, coordinated initiatives and oversight that do not currently exist. In this AMIA position paper, the authors describe current and emerging challenges to the safe use of Adaptive CDS and lay out recommendations for the effective management and monitoring of Adaptive CDS.

Comparison of an oncology clinical decision-support system's recommendations with actual treatment decisions.

OBJECTIVE: IBM(R) Watson for Oncology (WfO) is a clinical decision-support system (CDSS) that provides evidence-informed therapeutic options to cancer-treating clinicians. A panel of experienced oncologists compared CDSS treatment options to treatment decisions made by clinicians to characterize the quality of CDSS therapeutic options and decisions made in practice. METHODS: This study included patients treated between 1/2017 and 7/2018 for breast, colon, lung, and rectal cancers at Bumrungrad International Hospital (BIH), Thailand. Treatments selected by clinicians were paired with therapeutic options presented by the CDSS and coded to mask the origin of options presented. The panel rated the acceptability of each treatment in the pair by consensus, with acceptability defined as compliant with BIH's institutional practices. Descriptive statistics characterized the study population and treatment-decision evaluations by cancer type and stage. RESULTS: Nearly 60% (187) of 313 treatment pairs for breast, lung, colon, and rectal cancers were identical or equally acceptable, with 70% (219) of WfO therapeutic options identical to, or acceptable alternatives to, BIH therapy. In 30% of cases (94), 1 or both treatment options were rated as unacceptable. Of 32 cases where both WfO and BIH options were acceptable, WfO was preferred in 18 cases and BIH in 14 cases. Colorectal cancers exhibited the highest proportion of identical or equally acceptable treatments; stage IV cancers demonstrated the lowest. CONCLUSION: This study demonstrates that a system designed in the US to support, rather than replace, cancer-treating clinicians provides therapeutic options which are generally consistent with recommendations from oncologists outside the US.

Leveraging conversational technology to answer common COVID-19 questions.

The rapidly evolving science about the Coronavirus Disease 2019 (COVID-19) pandemic created unprecedented health information needs and dramatic changes in policies globally. We describe a platform, Watson Assistant (WA), which has been used to develop conversational agents to deliver COVID-19 related information. We characterized the diverse use cases and implementations during the early pandemic and measured adoption through a number of users, messages sent, and conversational turns (ie, pairs of interactions between users and agents). Thirty-seven institutions in 9 countries deployed COVID-19 conversational agents with WA between March 30 and August 10, 2020, including 24 governmental agencies, 7 employers, 5 provider organizations, and 1 health plan. Over 6.8 million messages were delivered through the platform. The mean number of conversational turns per session ranged between 1.9 and 3.5. Our experience demonstrates that conversational technologies can be rapidly deployed for pandemic response and are adopted globally by a wide range of users.

Evaluating artificial intelligence in medicine: phases of clinical research.

Increased scrutiny of artificial intelligence (AI) applications in healthcare highlights the need for real-world evaluations for effectiveness and unintended consequences. The complexity of healthcare, compounded by the user- and context-dependent nature of AI applications, calls for a multifaceted approach beyond traditional in silico evaluation of AI. We propose an interdisciplinary, phased research framework for evaluation of AI implementations in healthcare. We draw analogies to and highlight differences from the clinical trial phases for drugs and medical devices, and we present study design and methodological guidance for each stage.

Evaluation of an artificial intelligence clinical trial matching system in Australian lung cancer patients.

OBJECTIVE: The objective of this technical study was to evaluate the performance of an artificial intelligence (AI)-based system for clinical trials matching for a cohort of lung cancer patients in an Australian cancer hospital. METHODS: A lung cancer cohort was derived from clinical data from patients attending an Australian cancer hospital. Ten phases I-III clinical trials registered on clinicaltrials.gov and open to lung cancer patients at this institution were utilized for assessments. The trial matching system performance was compared to a gold standard established by clinician consensus for trial eligibility. RESULTS: The study included 102 lung cancer patients. The trial matching system evaluated 7252 patient attributes (per patient median 74, range 53-100) against 11 467 individual trial eligibility criteria (per trial median 597, range 243-4132). Median time for the system to run a query and return results was 15.5 s (range 7.2-37.8). In establishing the gold standard, clinician interrater agreement was high (Cohen's kappa 0.70-1.00). On a per-patient basis, the performance of the trial matching system for eligibility was as follows: accuracy, 91.6%; recall (sensitivity), 83.3%; precision (positive predictive value), 76.5%; negative predictive value, 95.7%; and specificity, 93.8%. DISCUSSION AND CONCLUSION: The AI-based clinical trial matching system allows efficient and reliable screening of cancer patients for clinical trials with 95.7% accuracy for exclusion and 91.6% accuracy for overall eligibility assessment; however, clinician input and oversight are still required. The automated system demonstrates promise as a clinical decision support tool to prescreen a large patient cohort to identify subjects suitable for further assessment.

Artificial intelligence-based conversational agent to support medication prescribing.

OBJECTIVE: This article describes the system architecture, training, initial use, and performance of Watson Assistant (WA), an artificial intelligence-based conversational agent, accessible within Micromedex®. MATERIALS AND METHODS: The number and frequency of intents (target of a user's query) triggered in WA during its initial use were examined; intents triggered over 9 months were compared to the frequency of topics accessed via keyword search of Micromedex. Accuracy of WA intents assigned to 400 queries was compared to assignments by 2 independent subject matter experts (SMEs), with inter-rater reliability measured by Cohen's kappa. RESULTS: In over 126 000 conversations with WA, intents most frequently triggered involved dosing (N = 30 239, 23.9%) and administration (N = 14 520, 11.5%). SMEs with substantial inter-rater agreement (kappa = 0.71) agreed with intent mapping in 247 of 400 queries (62%), including 16 queries related to content that WA and SMEs agreed was unavailable in WA. SMEs found 57 (14%) of 400 queries incorrectly mapped by WA; 112 (28%) queries unanswerable by WA included queries that were either ambiguous, contained unrecognized typographical errors, or addressed topics unavailable to WA. Of the queries answerable by WA (288), SMEs determined 231 (80%) were correctly linked to an intent. DISCUSSION: A conversational agent successfully linked most queries to intents in Micromedex. Ongoing system training seeks to widen the scope of WA and improve matching capabilities. CONCLUSION: WA enabled Micromedex users to obtain answers to many medication-related questions using natural language, with the conversational agent facilitating mapping to a broader distribution of topics than standard keyword searches.

The Value of the Surgeon Informatician.

Clinical informatics is an interdisciplinary specialty that leverages big data, health information technologies, and the science of biomedical informatics within clinical environments to improve quality and outcomes in the increasingly complex and often siloed health care systems. Core competencies of clinical informatics primarily focus on clinical decision making and care process improvement, health information systems, and leadership and change management. Although the broad relevance of clinical informatics is apparent, this review focuses on its application and pertinence to the discipline of surgery, which is less well defined. In doing so, we hope to highlight the importance of the surgeon informatician. Topics covered include electronic health records, clinical decision support systems, computerized order entry, data analytics, clinical documentation, information architectures, implementation science, quality improvement, simulation, education, and telemedicine. The formal pathway for surgeons to become clinical informaticians is also discussed.

Deep learning predicts extreme preterm birth from electronic health records.

OBJECTIVE: Models for predicting preterm birth generally have focused on very preterm (28-32 weeks) and moderate to late preterm (32-37 weeks) settings. However, extreme preterm birth (EPB), before the 28th week of gestational age, accounts for the majority of newborn deaths. We investigated the extent to which deep learning models that consider temporal relations documented in electronic health records (EHRs) can predict EPB. STUDY DESIGN: EHR data were subject to word embedding and a temporal deep learning model, in the form of recurrent neural networks (RNNs) to predict EPB. Due to the low prevalence of EPB, the models were trained on datasets where controls were undersampled to balance the case-control ratio. We then applied an ensemble approach to group the trained models to predict EPB in an evaluation setting with a nature EPB ratio. We evaluated the RNN ensemble models with 10 years of EHR data from 25,689 deliveries at Vanderbilt University Medical Center. We compared their performance with traditional machine learning models (logistical regression, support vector machine, gradient boosting) trained on the datasets with balanced and natural EPB ratio. Risk factors associated with EPB were identified using an adjusted odds ratio. RESULTS: The RNN ensemble models trained on artificially balanced data achieved a higher AUC (0.827 vs. 0.744) and sensitivity (0.965 vs. 0.682) than those RNN models trained on the datasets with naturally imbalanced EPB ratio. In addition, the AUC (0.827) and sensitivity (0.965) of the RNN ensemble models were better than the AUC (0.777) and sensitivity (0.819) of the best baseline models trained on balanced data. Also, risk factors, including twin pregnancy, short cervical length, hypertensive disorder, systemic lupus erythematosus, and hydroxychloroquine sulfate, were found to be associated with EPB at a significant level. CONCLUSION: Temporal deep learning can predict EPB up to 8 weeks earlier than its occurrence. Accurate prediction of EPB may allow healthcare organizations to allocate resources effectively and ensure patients receive appropriate care.